MathSymb performance measurements
overview
want to understand this report?
| mathSimplify (False) | |
| mathSimplify (True) | |
| mathExpand (False) | |
| mathExpand (True) |
mathSimplify (False)/ExprAF2
9.4 9.5 9.60 9.70 9.8 9.9
mean |
4 6 8 2×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 9.60 μs | 9.65 μs | 9.70 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 9.63 μs | 9.66 μs | 9.72 μs |
| Standard deviation | 77.7 ns | 104 ns | 134 ns |
Outlying measurements have slight (5.8%) effect on estimated standard deviation.
mathSimplify (False)/ExprAF3
27.8 27.9 28.0 28.1 28.2 28.3
mean |
1000 1500 2000 2500 3000 500 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 28.0 μs | 28.0 μs | 28.1 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 28.0 μs | 28.0 μs | 28.1 μs |
| Standard deviation | 112 ns | 145 ns | 181 ns |
Outlying measurements have no (0.8%) effect on estimated standard deviation.
mathSimplify (False)/ExprAF4
29 30 31 28.5 29.5 30.5 31.5
mean |
1000 1500 2000 2500 3000 500 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 29.1 μs | 29.2 μs | 29.5 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 29.0 μs | 29.2 μs | 29.7 μs |
| Standard deviation | 296 ns | 609 ns | 1.02 μs |
Outlying measurements have moderate (16.2%) effect on estimated standard deviation.
mathSimplify (False)/ExprAF5
93 94 95 93.5 94.5 95.5
mean |
400 600 800 1000 200 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 93.7 μs | 93.9 μs | 94.2 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 94.0 μs | 94.3 μs | 94.5 μs |
| Standard deviation | 659 ns | 748 ns | 925 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
mathSimplify (False)/ExprAF6
115 115 116 116 117 117 118
mean |
200 300 400 500 600 700 100 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 115 μs | 116 μs | 116 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 116 μs | 116 μs | 116 μs |
| Standard deviation | 626 ns | 754 ns | 903 ns |
Outlying measurements have slight (1.0%) effect on estimated standard deviation.
mathSimplify (False)/ExprAF7
212 213 214 215 216 217 218
mean |
100 150 200 250 300 350 400 50 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 213 μs | 214 μs | 215 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 213 μs | 214 μs | 214 μs |
| Standard deviation | 1.06 μs | 1.42 μs | 1.86 μs |
Outlying measurements have slight (1.2%) effect on estimated standard deviation.
mathSimplify (False)/ExprAF8
366 367 368 369 370 371
mean |
100 150 200 250 50 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 369 μs | 369 μs | 370 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 368 μs | 369 μs | 369 μs |
| Standard deviation | 1.20 μs | 1.48 μs | 1.91 μs |
Outlying measurements have slight (1.4%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF2
12.5 12.6 12.6 12.7 12.7 12.8 12.8 12.9 12.9 13.0
mean |
2 3 4 5 6 7 1×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 12.6 μs | 12.7 μs | 12.7 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 12.7 μs | 12.7 μs | 12.8 μs |
| Standard deviation | 104 ns | 122 ns | 149 ns |
Outlying measurements have no (0.7%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF3
23 22.4 22.5 22.6 22.7 22.8 22.9
mean |
1000 1500 2000 2500 3000 3500 4000 500 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 22.6 μs | 22.6 μs | 22.7 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 22.6 μs | 22.7 μs | 22.7 μs |
| Standard deviation | 120 ns | 144 ns | 194 ns |
Outlying measurements have no (0.8%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF4
65 64.3 64.5 64.8 65.3 65.5
mean |
500 750 1000 1250 250 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 64.7 μs | 64.8 μs | 64.9 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 64.7 μs | 64.8 μs | 65.0 μs |
| Standard deviation | 180 ns | 275 ns | 367 ns |
Outlying measurements have no (0.9%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF5
96 97 95.5 96.5 97.5
mean |
400 600 800 200 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 96.5 μs | 96.8 μs | 97.0 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 96.4 μs | 96.6 μs | 96.7 μs |
| Standard deviation | 438 ns | 519 ns | 609 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF6
145 146 147 148 149 150 151 152
mean |
200 300 400 500 600 100 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 147 μs | 147 μs | 148 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 146 μs | 147 μs | 148 μs |
| Standard deviation | 669 ns | 1.32 μs | 1.98 μs |
Outlying measurements have slight (1.1%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF7
212 214 216 218 220
mean |
100 150 200 250 300 350 400 50 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 214 μs | 215 μs | 216 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 214 μs | 215 μs | 216 μs |
| Standard deviation | 1.41 μs | 1.90 μs | 2.71 μs |
Outlying measurements have slight (1.2%) effect on estimated standard deviation.
mathSimplify (True)/ExprAF8
455 458 460 463 465 468
mean |
100 150 200 50 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 458 μs | 459 μs | 460 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 458 μs | 458 μs | 460 μs |
| Standard deviation | 1.87 μs | 2.77 μs | 3.98 μs |
Outlying measurements have slight (1.4%) effect on estimated standard deviation.
mathExpand (False)/ExprAF2
540 550 560 570 580 590 600 610 620 630
mean |
50 75 100 125 150 25×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 587 ns | 592 ns | 597 ns |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 585 ns | 591 ns | 599 ns |
| Standard deviation | 14.3 ns | 19.4 ns | 24.6 ns |
Outlying measurements have moderate (44.8%) effect on estimated standard deviation.
mathExpand (False)/ExprAF3
3 2.9 2.92 2.95 2.98 3.02
mean |
10 15 20 25 30 5×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.93 μs | 2.95 μs | 2.96 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 2.94 μs | 2.95 μs | 2.97 μs |
| Standard deviation | 32.7 ns | 39.1 ns | 47.5 ns |
Outlying measurements have moderate (10.0%) effect on estimated standard deviation.
mathExpand (False)/ExprAF4
6.3 6.35 6.4 6.45 6.5 6.55 6.6
mean |
4 6 8 10 12 14 2×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 6.47 μs | 6.51 μs | 6.54 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 6.46 μs | 6.49 μs | 6.51 μs |
| Standard deviation | 62.3 ns | 74.1 ns | 96.9 ns |
Outlying measurements have slight (6.7%) effect on estimated standard deviation.
mathExpand (False)/ExprAF5
13.8 13.9 14.0 14.1 14.2 14.3 14.4 14.5
mean |
2 3 4 5 6 1×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.0 μs | 14.0 μs | 14.1 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 14.1 μs | 14.1 μs | 14.2 μs |
| Standard deviation | 123 ns | 169 ns | 226 ns |
Outlying measurements have slight (6.8%) effect on estimated standard deviation.
mathExpand (False)/ExprAF6
30 31 30.5 31.5
mean |
1000 1500 2000 2500 500 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 30.4 μs | 30.6 μs | 31.0 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 30.5 μs | 30.6 μs | 30.8 μs |
| Standard deviation | 329 ns | 433 ns | 578 ns |
Outlying measurements have slight (8.5%) effect on estimated standard deviation.
mathExpand (False)/ExprAF7
66 67 68 69 70 66.5 67.5 68.5 69.5
mean |
400 600 800 1000 1200 200 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 67.5 μs | 67.9 μs | 68.4 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 67.7 μs | 68.0 μs | 68.3 μs |
| Standard deviation | 654 ns | 931 ns | 1.20 μs |
Outlying measurements have slight (7.1%) effect on estimated standard deviation.
mathExpand (False)/ExprAF8
145 146 147 148 149 150 151 152 153
mean |
200 300 400 500 600 100 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 147 μs | 148 μs | 149 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 148 μs | 149 μs | 149 μs |
| Standard deviation | 1.92 μs | 2.26 μs | 2.79 μs |
Outlying measurements have slight (7.3%) effect on estimated standard deviation.
mathExpand (True)/ExprAF2
625 650 675 700 725 750 775
mean |
40 60 80 100 120 140 20×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 664 ns | 675 ns | 690 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 664 ns | 676 ns | 691 ns |
| Standard deviation | 27.2 ns | 35.8 ns | 49.7 ns |
Outlying measurements have severe (67.8%) effect on estimated standard deviation.
mathExpand (True)/ExprAF3
3.2 3.25 3.30 3.35
mean |
10 15 20 25 5×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.27 μs | 3.29 μs | 3.31 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.26 μs | 3.28 μs | 3.30 μs |
| Standard deviation | 35.0 ns | 41.9 ns | 55.3 ns |
Outlying measurements have slight (9.0%) effect on estimated standard deviation.
mathExpand (True)/ExprAF4
7 7.10 7.2 7.30 7.4
mean |
4 6 8 10 12 2×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.19 μs | 7.24 μs | 7.29 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 7.17 μs | 7.23 μs | 7.26 μs |
| Standard deviation | 86.1 ns | 106 ns | 136 ns |
Outlying measurements have moderate (10.6%) effect on estimated standard deviation.
mathExpand (True)/ExprAF5
16.2 16.3 16.4 16.5 16.6 16.7 16.8
mean |
2 3 4 5 1×10³ iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 16.4 μs | 16.5 μs | 16.5 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 16.5 μs | 16.5 μs | 16.6 μs |
| Standard deviation | 130 ns | 146 ns | 184 ns |
Outlying measurements have no (0.7%) effect on estimated standard deviation.
mathExpand (True)/ExprAF6
34 35 36 34.5 35.5
mean |
1000 1500 2000 2500 500 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 34.9 μs | 35.1 μs | 35.2 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 34.9 μs | 35.0 μs | 35.2 μs |
| Standard deviation | 303 ns | 385 ns | 523 ns |
Outlying measurements have no (0.8%) effect on estimated standard deviation.
mathExpand (True)/ExprAF7
74 76 78 80 82 84 86
mean |
400 600 800 1000 200 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 76.8 μs | 77.4 μs | 79.0 μs |
| R² goodness-of-fit | 0.997 | 0.998 | 0.998 |
| Mean execution time | 78.1 μs | 79.2 μs | 80.6 μs |
| Standard deviation | 2.79 μs | 3.18 μs | 4.05 μs |
Outlying measurements have moderate (38.3%) effect on estimated standard deviation.
mathExpand (True)/ExprAF8
160 162 164 166 168
mean |
200 300 400 500 100 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 162 μs | 163 μs | 164 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 162 μs | 163 μs | 164 μs |
| Standard deviation | 1.39 μs | 1.95 μs | 2.60 μs |
Outlying measurements have slight (1.1%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.